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Abstract 

MINERz/A (Main INjector ExpeRiment v-K) is a new few-GeV neutrino 
cross section experiment that began taking data in the FNAL NuMI (Fermi 
National Accelerator Laboratory Neutrinos at the Main Injector) beam- line 
in March of 2010. MINERz/A employs a fine-grained scintillator detector 
capable of complete kinematic characterization of neutrino interactions. This 
paper describes the MINERz/A data acquisition system (DAQ) including the 
read-out electronics, software, and computing architecture. 
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1 1. Introduction 

2 MINERz/A (Main INjector ExpeRiment z/-A) is a low-to-medium energy 

3 neutrino- nucleus scattering experiment at FNAL (Fermi National Accelerator 

4 Laboratory). It utilizes the high-intensity NuMI (Neutrinos at the Main 

5 Injector) beamline and sits in the MINOS (Main Injector Neutrino Oscillation 

6 Search) Near Detector Hall, directly in front of the MINOS Near Detector 

7 and 105 meters below grade. MINERz/A does not extend far enough in the 

8 beam direction to range out muons with momentum over about 2 GeV/c, 

9 so the MINOS Near Detector is directly employed as a muon spectrometer. 

10 The references contain more detailed information on the physics program 

11 and general details of MINERz/A^ MINOS ^ and the NuMI beam^ 

12 This document discusses the MINERz/A data acquisition system (DAQ) 

13 in two steps. First, the custom electronics are described, followed by a de- 

14 script ion of the DAQ software and computing architecture. 

15 The initial specifications for the DAQ readout system asked for a readout 

16 rate of ^100 kB/s with a duty factor defined by one ^ 10 microsecond beam 

17 spill every 2.2 seconds^. Fortunately, those specifications were surpassed as 

18 our initial estimate for required data throughput was low by about a factor 

19 of five. 
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20 2. Overview of the Readout Scheme in MINERi/A 

21 The DAQ, slow controls (SC), and near-online monitoring ("nearline") 

22 systems are built around MINERz/A-specific readout electronics. MINERz/A 

23 utilizes plastic scintillator as its fundamental detector technology. Light 

24 is delivered via wavelength shifting fiber to Hamamatsu R7600 64-channel 

25 multi- anode photomultiplier tubes (PMTs) ^. A small number of single- anode 

26 PMTs are also mounted for a "Veto Wall" detector that sits in front of the 

27 main MINERz/A detector. 

28 Each PMT is read out by a Front End Board (FEB)- mounting six 

29 Application-Specific Integrated Circuit (ASIC) chips (TriP-t chips) that dig- 
so itize and store charge using pipeline ADCs. Input charges from the PMT an- 

31 odes are divided into high, medium, and low gain channels using a capacitive 

32 divider to increase the dynamic range. The high gain is 1.25 fC/ADC, the 

33 medium is 4 fC/ADC, and the low is 15.6 fC/ADC. The FEBs generate the 

34 high voltage for most of the PMTs using an on-board Cockroft- Walton (CW) 

35 generator. The control circuitry resides on the FEB, while the CW chain it- 

36 self resides on the base, with appropriate taps for each dynode. A small 

37 number of the PMTs are single-anode and use resistor-base technology for 

38 voltage control. These PMTs are used to read out the Veto Wall and employ 

39 a separate voltage control system discussed below. All FEB operations are 

40 controlled by a Spartan 3E Field-Programmable Gate Array (FPGA) chip. 

41 The FPGAs decode timing signals received over the unshielded twisted pair 

42 (UTP) cables, sequence the TriP-t chips and decode and respond appropri- 

43 ately to communication frames received over the data link. The FPGA also 

44 controls the CW and other aspects of FEB operation. For data collection, 

45 the boards are daisy-chained together (into "chains") using standard UTP 

46 ethernet networking cables with a custom protocol and Low Voltage DiflFer- 

47 ential Signaling (LVDS). Of the four pairs in the cable, one is dedicated to 

48 timing, including clock and encoded signals, one is dedicated for data, one 

49 is used to indicate the sync-lock status of the data Serializer/Deserializer 

50 (SERDES) and one for a test pulse. 

51 The readout chain is connected at both ends to a custom VME mod- 

52 ule called the Chain ReadOut Controller (CROC)-. A CROC supports four 

53 Front End (FE) channels, each serving one chain of up to fifteen FEBs, 

54 though no chain is longer than ten FEBs in MINERz/A. Each of the four 

55 CROC channels contains a 6 kB dual-port memory for storing messages 

56 (called "frames") from the FEBs. The CROCs in turn receive timing and 
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57 trigger commands from another custom VME module, the CROC Interface 

58 Module (CRIM)^, each of which distributes timing to up to four CROCs. 

59 MINERz/A uses two VME crates, each containing a CAEN V2718 crate con- 

60 troUer^, two CRIMs, and roughly a half-dozen CROCs. There is no hit-based 

61 trigger but rather a timing-based integration gate synchronized to the FNAL 

62 Main Injector (MI) timing signals. Finally, the CRIMs receive information 

63 from the timing system of the MINOS experiment (used for event match- 

64 ing) and the state of the MI via the MINERz/A Timing Module (MvTM)^^. 

65 The MvTM does not have a VME interface, but uses the VME crate for 

66 power. The start time in MINOS detector timing coordinates is logged as 

67 a 28-bit number in the DAQ Header bank data. See Table [1] for a com- 

68 plete module and card accounting. Only one MvTM is required to service all 

69 timing needs. Each individual crate is readout by a computer (PC) mount- 

70 ing a CAEN A2818 PCI card^^ that communicates with the crate controller 

71 module via optical cable. 



Era 


VME Crates 


CROCs 


FEBs 


Total Channels 


Fall 2009 


2 


8 


272 


17,408 


Spring 2010 


2 


14 


491 


31,424 


Summer 2010 + 


2 


15 


509 


32,576 



Table 1: MINERi/A read-out hardware installations. 

72 An integration gate on the FEBs is opened synchronously with the de- 

73 livery of neutrino beam spills. Beam spills delivered by the MI are approx- 

74 imately 10 microseconds long, and are delivered every 2.2 or 2.06 seconds, 

75 depending on the MI operating mode. Our readout gate opens 500 nanosec- 

76 onds before the arrival of the spill and remains open 5.5 microseconds af- 

77 ter the end. During normal data-taking, one additional calibration gate is 

78 recorded between neutrino spills, either a random sampling of the detector 

79 noise (a pedestal gate) or in coincidence with LEDs flashing the PMTs (a 

80 "light-injection" gate). Pedestals are used to establish baseline readout lev- 

81 els and light-injection to monitor gain drifts in the PMTs by studying the 

82 photo-statistics of the LED signals. 

83 At the end of the gate, the readout system runs a software loop over all 

84 of the FEBs and collects data in the form of device "frames" (well-deflned 

85 and formatted byte data with attached headers) one at a time for the state 

86 of the high-voltage, hit timing, and flnally the hit blocks themselves. Each 
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87 frame is passed through a chain to a CROC FE channel where it is stored 

88 briefly before being passed to a readout computer for archival and monitoring 

89 processes. 

90 Readout of the electronics is done in pairs of nested loops. Requests for 

91 data are sent to the first FEB on each FE channel before looping back to 

92 retrieve the data from the CROC memory. Then the next set of FEBs are 

93 handled in another dual loop over FE channels, etc. This allows the DAQ 

94 computers to communicate with the next CROC while the previous one is 

95 fetching data from the FEBs on one of its readout chains. 

96 In normal data-taking MINERz/A reads out in a "zero suppression" mode. 

97 In this mode, the DAQ first reads the discriminator frame data for each 

98 board and stores frames that contain hits above the discriminator threshold 

99 (about 70 fC). It then loops over those boards and reads the FPGA registers 

100 and all time-stamped ADC hit block frames. The FEBs are configured to 

101 enable readout of up to eight ADC blocks: seven time-stamped hits and 

102 one "un-timed" hit at the end of the integration gate. The un-timed hit 

103 carries no discriminator timing information and contains all the integrated 

104 charge between the last timed hit and the end of the gate. The FPGA 

105 registers contain configuration data, high-voltage read-back, and the local 

106 ''gate time stamp" - the time the gate was opened for integration in the local 

107 FEB counter time coordinates. Boards without hits above threshold are not 

108 read out in this loop and neither are un-time-stamped ADC frames (charge 

109 collected below threshold and stored until the end of the integration gate), 
no All seven available time-stamped hits are read out. During construction 

111 and early low-energy configuration running, a configuration with five time- 

112 stamped hits and one "un-timed" end-of-gate hit was used. See Table [2] for 

113 a listing of the sizes of each data type. 

114 Despite using more than 32,000 active channels in MINERz/A, the quan- 

115 tity of data per neutrino spill is not large due to low interaction cross-sections. 

116 With a 2.2 second MI cycle and 35 x 10^^ protons on target per spill, roughly 

117 1 MB per spill for the entire detector in zero-suppression mode running in 

118 the NuMI low-energy configuration is collected, where the most frequent neu- 

119 trino energy is ^3 GeV. About half of those data are beam spill data and 

120 half are calibration data. 

121 Readout time for a single frame is approximately 500 microseconds. Most 

122 of that time is in message preparation and in setting up a block transfer; for 

123 our data the readout speeds per frame are largely insensitive to the size of the 

124 frame. Physics gate readouts typically require 1200 frames on average, while 
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Frame Type 


Size (bytes) 


Comments 


FPGA Programming 


66 


Spartan 3E control registers. 


ADC 


444 


Hit data. 


Discriminator 


16 + 40 per 
TriP-t per hit 


Time stamps for each hit. 


TriP Programming 


652 (read) 
758 (write) 


TriP-t internal registers. 


DAQ Header 


48 


Not a true "frame." 
End of gate marker. 



Table 2: Frame sizes per FEB for MINERi^A data objects. The FPGA programming 
frames track configuration details and provide data on the high voltage, gate initiation 
time, and some useful error-checking bits. The TriP-t chips were originally designed for 
the DO experiment and so carried timing data that is not usable by MINERz/A. These 
data were trimmed in the packing algorithm for more efficient use of space. This reduced 
storage bloat, but did not meaningfully impact speed. Only discriminator frames are of 
variable size. The TriP-t programming frames are not read or written to during data 
acquisition. They are only accessed during configuration stages by the slow controls. 

125 calibration gate readouts require 1600 (channel occupancy is lower in physics 

126 gates). Therefore, reading a typical "cycle" (single physics gate plus single 

127 calibration gate) requires 1.4 seconds. Calibration readout times are very 

128 stable, but physics gate readout times vary with the level of activity in the 

129 detector. The DAQ is programmed to protect the integrity of physics gates 

130 above all else. It will only attempt a calibration readout in the case where 

131 the physics readout is accomplished in under 0.9 seconds. If the subsequent 

132 calibration readout extends beyond 0.9 seconds, the DAQ will dump that 

133 readout and simply arm for the next physics spill. However, the DAQ will 

134 not interrupt readout of a physics gate. Propagation of the start gate signal 

135 is blocked in hardware and only unlatched after readout is complete. 

136 3. System- Wide Timing Signals 

137 Time-stamps are recorded for the MINERz/A experiment using a single 

138 53.1 MHz crystal on the MvTM which is distributed to all of the electronics 

139 components - from MvTM to the CRIMs, from the GRIM to the CROCs, 

140 from the CROCs to the FEBs. See Fig. [1] for an illustration of the timing 

141 signals loop. Each board regenerates the clock using phase-locked loop (PLL) 
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142 
143 
144 
145 

146 
147 
148 
149 



circuits. The 53.1 MHz clock is distributed on the readout cable, but the 
FEBs use the onboard PLL to multiply the frequency by two. All times in 
MINERz/A are defined in units of system ticks, or clock ticks. A clock tick 
is equal to 

1// = 1/(2 X 53.1 X 10^ Hz) = 9.4ns (1) 

Additionally the FPGA on the FEB uses a quadrature circuit, providing a 
granularity of ^2.4 ns for the discriminator time stamps. Throughout the 
remainder of this document, the term "clock tick" will refer to the unit used 
by the FEBs (^9.4 ns) unless explicitly noted otherwise. 




Figure 1: The DAQ Loop Timing Signals.— 



150 The FEBs, CROCs and CRIMs can operate using on board oscillators, 

151 but FEBs and CROCs in MINERz/A are always operated by synchronizing 

152 their clocks to a radio frequency (RF) signal provided by the CRIMs. The 

153 CRIMs are typically operated in "MvTM Mode," and locked to the MvTM 

154 clock, but in certain situations are placed in "Internal Mode," where the 

155 clock is supplied by the internal crystal oscillator. See Table [3l MINERz/A 

156 electronics were designed to operate without a trigger, so the cosmic and 

157 Test-Beam modes are special. In those cases, a coincidence between a live 

158 gate 16 /is wide) and an external logic pulse is searched for in order to 

159 make readout decisions. Because of dead-time between gates, the live-time 

160 is low in those operating modes 10%). 
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Data- Taking Mode 


CRIM Clock Mode 


Pedestal 


Internal 


Cosmic Rays 


Internal 


FTBF (Test-Beam) Beam 


Internal 


NuMI Beam 


MvTM 


Light Injection (LI) 


MvTM 


Mixed Mode (Beam with LI or Pedestal) 


MvTM 



Table 3: CRIM timing modes for different data-taking modes. "FTBF" refers to the 
Fermilab Test Beam Facility. 

161 When operating in the internal timing mode, each CRIM supplies its own 

162 clock and either the DAQ system sends a new open-gate command after each 

163 successful readout or the CRIM opens the gates independently at a frequency 

164 set by the user ranging between 0.5 Hz and 52 kHz. The first sort of internal 

165 clock mode operation is used for standalone pedestal runs and the second 

166 requires an external trigger in coincidence to initiate readout (cosmic or test 

167 beam) . 

168 Encoded clock and trigger signals enter the CROCs over LVDS via RJ-45 

169 input connectors: 

170 1. RF - Radio Frequency 

171 2. SGATE - Start Gate (open an integration gate) 

172 3. CNRST - Counter Reset 

173 4. FCMND - Fast Command (special commands) 

174 5. TCALB - Timing Cahbration Pulse 

175 See Fig. [2] for the structure of the encoded commands on the clock line. 

176 In either Internal or MvTM timing mode, the CRIM can receive exter- 

177 nal signals via a set of LEMO connectors on its front panel or via an RJ-45 

178 connector. The MvTM is able to recieve timing signals from the MINOS 

179 Master Timing Module over UTP Cat5e using LVDS. The MvTM addition- 

180 ally can send and relay Low Voltage Transistor- Transistor Logic (LVTTL) 

181 signals through the pairs of in-and-out LEMO connectors. One LEMO in- 

182 put is used to receieve a LVTTL signal providing MI cycle status from the 

183 Accelerator Division (this signal ultimately detemines the start-gate time). 

184 The other two LEMO connectors are used to fire the light injection system 

185 (discussed later). 
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RF 

SGATE_H 
SGATE_L 

CNRST_H 
FCMND_H 
TCALB_H 

Figure 2: The Timing Encoder Signals.— 

186 During normal operation, the CRIM clock is derived from the MvTM. In 

187 principle, the MvTM can sync to the MINOS clock signal but it runs free 

188 against that clock and the MvTM internal oscillator is used to synchronize 

189 all of the clocks. The arrival time of the MINOS start-gate signal is recorded 

190 in a register in the CRIM, however. The MINOS and MINERz/A clocks are 

191 therefore running in sync with an unknown phase shift. When in MvTM 

192 mode, the CRIM will raise an interrupt when an external trigger pulse is 

193 detected - for example, when the spill signal comes from the MI. This inter- 

194 rupt begins the readout sequence and blocks until it is reset after readout is 

195 complete. Fig. [3] iUustrates the timing module logic. 

196 The CRIM interprets Start-gate (SGATE), trigger (TRIG), timing cali- 

197 bration (TCALB), and counter-reset (CNRST) LVTTL signals and generates 

198 corresponding output in a slightly different way for each of the different tim- 

199 ing modes. 



200 1. SGATE is sent through SGATE OUT in all timing modes. 

201 2. TRIG is sent only in MvTM mode, and only in case of a software 

202 command to open a gate. 

203 3. TCALB is sent synchronously with SGATE in most CRIM firmware 

204 versions. It can be configured to only send this signal when an external 

205 TRIG is supplied. 

206 4. CNRST is sent synchronously with SGATE in most CRIM firmware 

207 versions. The FEBs interpret this as a "Counter Load" signal and set 

208 their counters to a value specified in the FPGA programming registers. 

209 This is part of the timing synchronization scheme for the FEBs. 
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From DAQ loop module 



Figure 3: The GRIM (timing module) block diagram.— 
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210 The clock and data travel only in one direction over the readout chains- from 

211 the CROC to each of the FEBs in turn and arriving back at the CROC. The 

212 CROC checks that the clock is present at the receiving end and will set an 

213 error flag if it is not. Further data acquisition is not possible until this error 

214 is cleared. 



215 Our configuration for data-taking in the NuMI hall is: 

216 1. A LEMO cable from the accelerator division rack is attached to LEMO 

217 input 1 on the MvTM. A signal synchronized with the MI operational 

218 cycle arrives ^215 /xs before a spill. A delay is programmed into the 

219 FEB FPGA's to account for the offset. This signal is sent to the CRIM 

220 from the MvTM via a CAT3 cable, and propagated from there through 

221 the CROCs to the FEBs. The FEBs use only the rising edge of the 

222 SGATE signal to open a gate. The falling edge is ignored. 

223 2. A dedicated CAT3 cable is routed from the MINOS clock rack into 

224 the Master Clock Interface connector on the MvTM. This cable car- 
ries CNRST, TCALB, and SGATE encoded clock signals. The CRIM 

226 receives those signals as: 

• MINERz/A CNRST - MINOS CNRST. Set by the MINOS GPS 

228 clock to reset asynchronous to the other signals at a rate of 1 Hz. 

229 This signal is currently masked from the FEBs, but used to reset 

230 the counter on the CRIM. 

231 • MINERz/A TCALB - The MINOS SGATE signal is mapped here 

232 in the MvTM. The clock time is recorded (using on-board coun- 

233 ters) for the arrival of this signal and stored on a register in the 

234 CRIM that is read at the end of the acquisition cycle. 

235 • MINERz/A SGATE - This receiver is masked within the MvTM 

236 and the CRIM never sees a signal of this type. 

237 3. All of the CRIMs are daisy-chained together via the TRIG port, going 

238 out from the "Master CRIM" (the CRIM with the lowest address in 

239 the VME crate with the lowest address) and then in and out of all sub- 

240 sequent CRIMs (incrementing by CRIM address before crate address) 

241 until the input on the last CRIM (the CRIM with the highest address 

242 in the VME crate with the highest address) is reached. This chain is re- 

243 sponsible for propagating the start signal in light injection data taking. 

244 A software command is sent to the Master CRIM only and propagated 

245 to the others via this electrical connection. Additionally a signal is sent 
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246 from the Master CRIM SGATE connector to the hght injection box. 

247 This output is passed through the MvTM itself for shaping and delay. 

248 4. The CRIM is run in one of two timing modes (see Table [3j). In the 

249 Internal and MvTM modes a sequencer that sends SGATE, TRIG, 

250 and CNRST to the FEBs is employed. If running in "cosmic" mode for 

251 the MINERz/A Test Beam Experiment, TCALB is additionally sent if 

252 an interrupt was raised. 



253 4. Hit Timing 

254 The MINERz/A FEBs mount six TriP-t chips. Four are dedicated to the 

255 high and medium gain and two are dedicated to the low gain. Due to this 

256 division, the TriP-t chips must be grouped together when storing hits in order 

257 to avoid some potential ambiguities. This has important consequences for 

258 hit timing. 

259 4.1. TnP-t 'Tush'' Behavior 

260 Understanding hit storage is important for understanding hit timing. The 

261 TriP-t has a 48-cell deep analog pipeline and integrated charge is pushed into 

262 this pipeline when the discriminator fires according to a process described be- 

263 low. Each cell of charge corresponds to a "hit." This behavior is not internal 

264 to the TriP-tchip, but is controlled but the FPGA firmware. The readout 

265 depth (number of total allowed hits) is limited by the internal memory of 

266 the FPGA and the way digitized data from the hits is stored in the FPGA. 

267 There is fiexibility in the maximum number of hits that may be stored, but 

268 in standard configurations prior to 2012, MINERz/A stored up to eight hits. 

269 Charge is integrated over a time window set by the user. The whole 

270 window is referred to as a "gate." Activity is separated in time inside this 

271 gate by latching the system counter when the integrated charge goes over 

272 a user-defined threshold and then storing the charge integrated up to that 

273 time (this process is called "pushing the pipeline"). After storing the hit, 

274 the integrators are reset and re-opened. At the end of the gate, the analog 

275 information stored in the pipeline is digitized into RAM with one ADC block 

276 (or "hit") for each stored set of charges in the pipeline (for each "pipe"). 

277 The different times associated with these processes are tunable by the user. 

278 In MINERz/A, the standard gate length is 1702 clock ticks {^16 /xs). The 

279 integration time after a discriminator fires is 16 clock ticks (?^150 ns), and 

280 the reset requires 20 clock ticks (?^188 ns). 
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281 The six TriP-t chips on each FEB are numbered 0-5, and each carries 32 

282 channels. TriP-t chips 0-3 each operate 16 channels in the high gain mode 

283 and 16 channels in the medium gain mode. TriP-t chips 4 & 5 each operate 

284 32 channels in the low gain mode. There are 64 channels on each multi- 

285 anode PMT, so TriP-t chips 0, 1, and 4 serve one half of the PMT, and 

286 TriP-t chips 2, 3, and 5 serve the other. Note this means any given pixel on 

287 a PMT is serviced by two separate TriP-t chips - one chip with one channel 

288 operating in a low-gain mode (low amplification of signal), and one chip 

289 with two channels, one in a medium-gain mode and one in a high-gain mode 

290 (large amplification of signal). All three channels operate simultaneously and 

291 independently of each other, each recording a value proportional to the gain 

292 times the charge. In principle, every channel on a TriP-tchip supports a 

293 discriminator, but only the discriminators on the high gain TriP-t channels 

294 are active. This is because the high gain channels are the most sensitive 

295 and low and medium-gain channels cannot be over threshold without their 

296 corresponding high-gain channel also being over threshold. The channel map 

297 ties two high and medium-gain TriP-t chips to each low gain TriP-tchip. In 

298 order to associate data in the low gain with data in the high or medium 

299 for a given channel, the low-gain TriP-tfor that channel must be pushed 

300 with the high gain TriP-t chip. To avoid ambiguous hit assignment for the 

301 other channels serviced by the low-gain TriP-t chip that are not shared with 

302 the firing high-gain chip, the "parallel" high-gain chip must also be pushed. 

303 Thus, there are eflFectively two logical units, 32 channels in size, on each 

304 board that are used to fill the analog pipelines. 

305 4'^' Clock Ticks and Time 

306 Time on the FEBs is set relative to a free-running counter that counts 

307 up in clock ticks until it reaches the 32-bit unsigned integer maximum, at 

308 which point it simply rolls over to zero. The counters are reset to the value 

309 stored in the programming registers of the FPGA control chips when the 

310 FEBs receive a CNRST signal. By tuning this start value to account for 

311 signal propagation time, it is possible to synchronize the local counter values 

312 on the FEBs to within one clock tick. 

313 After a gate-open signal, the FEB loads its Timer register value and starts 

314 a counter that counts up to the 16-bit unsigned integer maximum. This delay 

315 is used to synchronize with beam activity and as a "quiet" period to blank 

316 charge pulses to the CW voltage system just before SGATE (at least 15 

317 microseconds is required). At the counter maximum the FEB gate opens 
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318 and the time on the free-running counter is latched as our gate start time. 

319 It is an offline software convention to subtract this time from aU subsequent 

320 calculations on a board. 

321 Because the low gain channels are shared across two high and medium 

322 gain sets of channels, when the discriminators fire charge is pushed and stored 

323 in groups of 32 channels. This means a hit in a given channel creates a 

324 timestamp Tq for the whole TriP-t chip via the high gain. That timestamp 

325 is further shared across the "TriP-t Chip Group" (0, 1, and 4 share the same 

326 base time, and 2, 3, and 5 share a separate base time). Subsequent hits 

327 within the 16-tick integration window have diflFering delay and quarter ticks. 

328 Crossing a discriminator threshold causes all three associated TriP-t chips to 

329 push 16 FEB clock ticks later, regardless of subsequent hits. 

330 When a discriminator fires, it sets its whole TriP-tchip group (0+1+4 

331 or 2+3+5) to push 20 system ticks later. The entire group shares the base 

332 integer time stamp for the earliest channel to fire on the group, and the 

333 initiating channel also stores a quarter-tick for fine resolution. The system 

334 clock continues to run and provides delay and quarter ticks to timestamp 

335 subsequent hits (prior to the push) to 1 /4 of a tick accuracy in other channels 

336 (pixels) in the group. 

337 4-3. Dead Time 

338 Coupling pairs of high gain TriP-t chips implies dead time impacts 32 

339 channels at a time. The reset after a push requires 20 clock ticks to complete. 

340 During the reset period, all three TriP-t chips that pushed are essentially dead 

341 to incoming signals. Hits during the reset period are un-timed and the charge 

342 recorded is either greatly attenuated or zero. For illustration, consider Fig. 

343 m When there is a hit on one half of a discriminator-active TriP-tpair at 

344 time To and another hit at a later time Ti, the second hit is latched in time 

345 as long as it arrives before the push. If the second hit on either pair of TriP- 

346 t chips arrives more than 16 ticks after the initiating hit, the later hit is not 

347 recorded. Time on both TriP-t chips is relative to Tq, the first threshold cross 

348 between the two. Fig. [5] shows a typical bank structure for a gate and Fig. 

349 [6] shows the memory structure of a discriminator bank. 

350 There is another form of dead time due to hit saturation. This occurs 

351 when the number of "pushes" allowed by the pipeline depth are exhausted. 

352 In this case, charge continues to integrate, but additional hits are no longer 

353 time-stamped. 
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First Hit, Second Hit, ^^^^ Reset 
Trip Trip 1 Complete 



"Dead" 



TO T1 TO + 1 6 ticks T1 + 1 6 ticks TO + -36 ticks 



Figure 4: A specific dead-time illustration on one FEB. Consider a hit on TriP-tO. This 
hit will set a timestamp, To here, and cause a pipeline push 16 clock ticks later for TriP- 
t chips 0, 1, and 4. A later hit at time Ti above does not initiate a new pipeline push and 
the time is stamped relative to Tq. The ADCs are effectively dead between Tq + 16 and 
To + 36 clock ticks. After that point, the ADCs on those TRiPs are fully live again and 
new hits over threshold will be timestamped. 



Discriminator Bank 
(Header Info with # of 
Hits per TriP-t) 


All hits for 
TriP-t 


HitO 


Hit 1 


All hits for 
TriP-t 1 


HitO 


Hitl 


All hits for 
TriP-t 2 


Hitl 


All hits for 
TriP-t 3 


Hitl 



Push Direction 
4- 


ADC Bank 
("Hit X") 


ADC Bank 
("Hit X+1") 


Garbage 
(Stale Data) 


TriP-t 
PipeDelay - Hit 


TriP-t 
PipeDelay - Hit 1 


Garbage 
(Stale Data) 


TriP-t 1 
PipeDelay - Hit 


TriP-t 1 
PipeDelay - Hit 1 


Garbage 
(Stale Data) 


Garbage 
(read anyway) 


TriP-t 2 
PipeDelay - Hit 1 


Garbage 
(Stale Data) 


Garbage 
(read anyway) 


TriP-t 3 
PipeDelay - Hit 1 



Figure 5: A generic example of the organization of data within the discriminator and 
ADC banks. The columns ("Hit X") denote hit RAM Functions - all of the data in the 
column is read at once and partial pipeline reads cannot be performed. In this case, there 
was a hit on either TriP-tO or 1, another independent hit on either TriP-t2 or 3, and 
a final hit on TriP-tO or 1. The ADCs cannot distinguish which TriP-tin the two pairs 
contained the initial hit for that pair. Again, this diagram does not distinguish which 
TriP-thad the channel hit first, and it cannot distinguish which of the hits in the "Hit 1" 
RAM Function was earlier - the discriminator bank must be decoded for that information. 
The end of integration "hit" is not included in the set of ADC banks shown here. It would 
appear in a new column titled "Hit 2" and would appear in all TriP-t 's. For each FEB 
read out, the following RAM Functions: ReadDiscr, ReadHitO, ReadHitT are looped 
through. 
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The discriminator block RAM for one TRiP 
is 16 bits wide (16 channels wide) and 20 
bits deep (1 6 delay ticks, a 2 bit quarter 
tick, and then the system timestamp). 



I I J i i i i j i J J i J i i i J 



Figure 6: A visualization of the discriminator bank for a single TriP-tchip. The system 
tick is stored in the 32 bits at the "bottom" of the figure. Quarter ticks for each channel 
are stored in the 32 bits above that at 2 bits per channel. Finally, the delay ticks are 
stored in 16 bits for each channel above the quarter ticks. 



354 4-4' FEB Timing Synchronization 

355 When the CRIM sends an open-gate command (SGATE) through the 

356 CROCs to the FEBs, the actual signal arrives at different times to the dif- 

357 ferent FEBs on the chain due to cable delays. This is accounted for in two 

358 steps: 

359 1. The FEB loads the value recorded in the Timer register of the FPGA. 

360 By staggering the time loaded to account for the finite signal propaga- 

361 tion time down the chain, all of the boards can synchronize the values 

362 on their local counters. 

363 2. Once the FEBs receive an open-gate signal they begin counting up 

364 to a pre-set open-gate time value, beginning not with zero, but with 

365 the Timer value mentioned in the previous step. This additional delay 

366 is meant to provide the PMT with time to settle into a stable high 

367 voltage in the case the Cockroft- Walton adjusted the voltage period 

368 just before the gate is opened. Once they reach this time, they open 

369 their integration gates. All the FEBs will physically open their gates 

370 at the same local time. 
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371 5. DAQ Software 

372 5.1. Readout Control 

373 The DAQ is a system built out of three computers running 64-bit Scientific 

374 Linux Fermi - one readout node for each VME crate and one master node. 

375 Data transfer between the readout nodes and master node is handled via the 

376 Event Transfer (ET) software package (part of CODA-^^), discussed further 

377 below. The head node initiates a run and creates a memory-mapped file to 

378 store data. It provides instructions to the readout nodes on when to begin 

379 the readout sequence. When readout begins, each of the slave nodes passes 

380 data frame-by-frame to the master for collation. At the end of a run, the head 

381 node converts the memory-mapped file into a fiat binary and releases it. One 

382 feature of this approach is that if the DAQ fails during readout, the collected 

383 data are already stored on disk and are not lost. The DAQ is written in C++ 

384 to run on Scientific Linux Fermi—, but should run with minimal modifications 

385 on most major Linux distributions. The code is divided into four libraries: 



386 1. Hardware access: Our hardware access libraries are built on top of a 

387 set of proprietary (closed-source) libraries provided by CAEN to access 

388 the V2718 VME crate controller^ using the A2818 PCI optical bridge 

389 card-^'. Functions from the libraries CAEN makes freely available for 

390 this hardware on its website are used to format command and control 

391 messages to the CROCs and CRIMs as required during readout. 

392 2. Event structure: Another set of libraries that define the types and 

393 structures of the MINERz/A data types. 

394 3. Acquisition control: These libraries and executables run the acquisition 

395 and interface with the Run Control software described in Section 15. 2[ 

396 4. Buffer management and interprocess communication (IPC): Like the 

397 hardware access libraries, the buffer management and IPC code uses 

398 external libraries. In particular, the open-source Event Transfer (ET) 

399 system version 9.0, based on CODA,^^ developed at the Thomas Jef- 

400 ferson Lab National Accelerator Facility (JLab)^'' is used. 

401 5.2. Run Control 

402 The run control is written in Python for fiexibility and platform-independence. 

403 There are two external Python libraries (in addition to the DAQ software, 

404 described in section 15.11) needed for various components of the run control to 

405 work: 
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406 1. wxPython-^^, chosen as our graphics toolkit because of its ease of use 

407 and maturity; 

408 2. pySerial^^, which provides a convenient RS-232 interface. 

409 6. Computing Architecture 

410 6.1. Overview of Data Handling 

411 MINERz/A uses two networks of server-class computers: one for physics 

412 data collection and one for data quality monitoring and run control. The 

413 data collection PCs are detailed in Section [71 For monitoring, MINERz/A 

414 uses two remote clusters of computers at Fermilab: one in the Feynman 

415 Computing Center (FCC), and the other at Wilson Hall (WH). These systems 

416 are discussed in Sections [8] and [9] 

417 Data are collected and logged underground independently of above-ground 

418 computing clusters. The DAQ can run without input from or a connection 

419 to any above-ground network with local storage for physics data, metadata, 

420 and logs sufficient for about a month of normal operations. The above- 

421 ground computers manage long-term data archival, log file metadata into 

422 SAM (Sequential Access Method - a metadata database application) pro- 

423 duce histograms for monitoring, and provide run control options. 

424 6.2. Data acquisition coordination 

425 The run control software manages connections between most of the com- 

426 puters used in the data acquisition and monitoring process such that they 

427 remain synchronized throughout. Kerberos- authenticated^^ SSH tunnels are 

428 used to connect clients (i.e., user shift consoles) to a master DAQ server pro- 

429 cess and the master DAQ server to the other slave machines, as described 

430 below. 

431 The heart of the run control is a central server process on the master 

432 DAQ node. It maintains communication with "dispatcher" processes running 

433 on each of the relevant slave remote nodes so as to instruct them to start 

434 or stop their various tasks as the state of data acquisition changes. It is 

435 also the entry point for communication from the graphical end client used 

436 by operators to interact with the DAQ. This "DAQ manager" process can 

437 coordinate data taking irrespective of the number of clients connected; once 

438 started, it will run through to completion of the data acquisition sequence 

439 even if the controlling client is disconnected. 
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440 The DAQ manager connects to a collection of remote nodes in various 

441 locations. Among these are readout nodes that insert the digitized data 

442 into the data stream and the monitoring nodes that run slow monitoring 

443 processing jobs and serve the data to the operator. 

444 Operators control the DAQ using a graphical client (also contained in the 

445 run control package) that connects to the DAQ manager over the network 

446 using a custom thin-client TCP protocol. This design accommodates both 

447 the Fermilab control room and collaborator institutions located around the 

448 world. 

449 7. Physical Organization of the Readout Equipment in NuMI 

450 All relevant MINERz/A electronics equipment is mounted in racks sitting 

451 on a deck beside the detector. All racks are equipped with smoke detection 

452 and interlocked power, with rack protection and slow monitoring built into 

453 the overall online scheme. 

454 Racks are labeled according to their primary hardware electronics com- 

455 ponents and there are four primary racks: 

456 1. The VME Rack (also sometimes called the DAQ Rack) 

457 2. The Light Injection (LI) Rack 

458 3. The Veto Rack 

459 4. The Spares Rack 

460 Our racks all consume less than 2500W each, meeting the FNAL require- 

461 ments for air cooling. See Fig. [7] for an overview of the arrangement of and 

462 connections between the three racks. 

463 7.L The VME Rack 

A6A The VME Rack contains the two VME crates that hold the MINERz/A 

465 readout electronics. It additionally mounts a pair of PCs and network sup- 

466 port components. Access control for devices that cannot be configured to use 

467 Kerberos is provided by a very restrictive ACL (Access Control List). See 

468 Fig. [8] for the rack layout and connection types. Insofar as the readout ar- 

469 chitecture is concerned, it contains (roughly in mechanical installation order 

470 from top to bottom): 

471 LA custom Temperature and Environment Monitor (TEM) device (la- 

472 beled the "Slow Monitor" ) that supports eight temperature probes that 
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Veto Rack 



Near Hall Overview 



Veto Wall FEB & Chassis 



LeCroy HV Control (4U) 



Master & Event Builder 
(1 U Linux PC) 
mnvonlinemaster.fnal.gov 



Upstream DAQ 
(3U Linux PC) 
mnvonlineO.fnal.gov 



Downstream DAQ 

(3U Linux PC) 
mnvonlinel .fnal.gov 



Switched PDU (1 U) 
minerva-pdu-2.fnal.gov 



2000W UPS (2U) 
mnv-vet-ups.fnal.gov 



LI Rack 



RPS (1U) 
minerva-rps-1 .fnal.gov 



PIN Box (1U) 



Light Injection Box 1 



Light Injection Box 



► VME Rack 



Environment Monitor 
minerva-env-1 .fnal.gov 



VME Crate 
(9U) 



VME Crate 1 
(9U) 



Switched PDU (1 U) 
minerva-pdu-1 .fnal.gov 



S-S-Minerva-1 
(Cisco Switch) 



Reserve PC 
(1U Linux PC) 
mnvonlinebckl .fnal.gov 



Veto Wall HV Control 
(1 U Windows PC) 
mnvonlinehvcontrol.fnal.gov 



2000W UPS (2U) 
mnv-vme-ups.fnal.gov 



Veto Wall 



Minerva 
Upstream 



Minerva 
Downstream 



Optical Cables & 
Readout 

Sensors 

Power 
General BNC or LEMO 



This ignores AC Distribution, FESB Power. Network connections are suppressed. 



Figure 7: Connection map for the MINERi^A readout system, ignoring AC distribution, 
FESB power, and safety systems. Network connections are also suppressed - every device 
with an internet name is hnked to the network switch housed in the VME rack. 
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473 may be deployed throughout the racks and over the detector surface. 

474 The device also logs humidity, pressure, and temperature at the end of 

475 six cabled sensors deployed on the detector. 

476 2. Two CAEN VME 8011 crates The crates are indexed as and 1. 

477 Crate contains a controller module, two CRIM modules, and eight 

478 CROC modules. Crate 1 contains a controller module, two CRIM 

479 modules, seven CROC modules, and a MvTM module. See Figs. [9] 

480 and do] for the VME module assignment in Crates and 1 respectively. 

481 3. An APC 120VAC Masterswitch III Switched Rack PDU (Power Distri- 

482 bution Unit), Model # AP7900. This PDU is a networked device that 

483 provides remote power control for the VME crates and all PCs in the 

484 rack. 

485 4. A 48 port Cisco C2960G switch, designated S-S-MINERVA-l.^^ 

486 5. Three 48VDC power supplies to power the FEB's (not pictured in the 

487 figures) . 

488 6. Three networked fuse panels for the 48VDC power supplies, networked 

489 to S-S-MINERVA-1. These fuse panels host a web interface with self- 

490 documented power control of the FESBs mounted on the MINERz/A 

491 detector (not pictured in the figures). 

492 7. A lU Linux PC serving as a back-up for the master DAQ node in the 

493 Veto Rack (see Section 17.31) . 

494 8. A lU Windows PC running control and monitoring software for the 

495 LeCroy High Voltage Power System (HVPS) located in the Veto Rack. 

496 Not currently installed. Space, power, and networking resources are 
mi reserved. 

498 9. An APC Smart-UPS 2200VA (2000W Un-interruptible Power Supply) 

499 supporting the switch and PCs. The primary function of this UPS is 

500 power filtering. In the event of a full power failure, the system will 

501 remain live for more than thirty minutes. 

502 1.2. The LI Rack 

503 The LI rack contains the MINERz/A custom-built Light Injection (LI) 

504 boxes ^i. The rack is intentionally kept sparse to provide sufficient space to 

505 route and store the optical fibers that connect from the back of the LI boxes 

506 to the PMTs mounted on top of MINERz/A. See Fig. [11] for the rack layout 

507 and connection types. For our purposes, it contains (roughly in mechanical 

508 installation order from top to bottom): 
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Figure 8: Connection map for the VME Rack. Power architecture is not shown, including 
the three networked fuse panels. 
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Figure 9: The connection and layout diagram for VME Crate 0. 
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Figure 10: The connection and layout diagram for VME Crate 1. The V1720 CAEN 
Waveform Digitizer is not currently installed. 
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509 1. A BiRa Rack Protection System (RPS). The RPS deploys a variety 

510 of monitoring cables into the VME Rack: voltage-monitoring cables 

511 to the VME crates, and airflow and temperature sensors. The RPS is 

512 connected to the S-S-MINERVA-1 switch located in the VME rack by 

513 ethernet. 

514 2. A PIN diode box. The PIN is optically connected to the LI box to 

515 monitor the stability of the LED light output. The PIN box is addi- 

516 tionally connected via BNC and specialized cables to a CAEN VI 720 

517 waveform digitizer mounted in VME Crate in the VME Rack. 

518 3. The primary LI box. This box contains control electronics and LEDs 

519 that service optical connectors on the back of the box. From those 

520 connectors light is piped via optical flber to the PMT boxes mounted 

521 on top of the detector. It is interfaced via a 9-pin serial RS-232 socket 

522 on the front of the box. This serial cable is connected to the MINERz/A 

523 DAQ PC cluster for conflguration commands. It additionally accepts 

524 a LEMO connection for the external trigger signal. 

525 4. A secondary (smaller) LI box to service the most forward detector 

526 modules. It uses the same electronics and connections as the larger 

527 box, but mounts fewer LEDs. 



528 7.3. The Veto Rack 

529 The Veto rack derives its name from the LeCroy High Voltage Power 

530 System mounted there that controls the high voltage on the resistor-base 

531 PMTs deployed to readout the "Veto Wall" sitting in front of the MINERz/A 

532 detector. It also contains the main readout PCs for the DAQ system. See 

533 Fig. [12] for the rack layout and connection types. For our purposes, its 

534 important contents are: 



535 1. Two MINERz/A FEBs attached to a special custom breakout board 

536 in a 2U chassis. The FEBs are standard MINERz/A boards, but the 

537 breakout board is required to interface the single-anode PMTs (resistive 

538 base) used to monitor the scintillator panels in the veto wall. 

539 2. A LeCroy Model HV4032A High Voltage Power System (HVPS). The 

540 HVPS connects to the readout PC located in the VME Rack via a 

541 special custom serial cable. It also connects to the Veto Wall PMTs. 

542 3. A lU Linux PC running the DAQ Master software. This PC is identical 

543 to the logger PC in the VME Rack. The DAQ Master node connects 

544 to the Slave Readout nodes via direct ethernet. 
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Near Hall Overview - The Light Injection (LI) Rack 



Serial (RS232) 
PC -► LI Box Directly 



LI Rack 
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This ignores Power Distribution & Smoke Detection. 

MINERvA Subnet; Optical Cables & Readout; 
Sensors; ^ Pbwer; General BNC or LEMO 



Figure 11: Connection map for the LI Rack. 
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545 4. Two 3U Linux PCs serving as DAQ Readout Slave nodes. Each PC 

546 mounts a CAEN a2818 PCI Optical Bridge card— and interfaces with 

547 one of the VME crates located in the VME Rack. 

548 5. An APC 120VAC Masterswitch III Switched Rack PDU (Power Distri- 

549 bution Unit), Model # AP7900. This PDU is a networked device that 

550 provides remote power control for the VME crates and all PCs in the 

551 rack. 

552 6. An APC Smart-UPS 2200VA (2000W Un-interruptible Power Supply) 

553 supporting the PCs. The primary function of this UPS is power filter- 

554 ing. In the event of a full power failure, the system will remain live for 

555 over thirty minutes. 
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Figure 12: Connection map for the Veto Rack. 
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556 7.^. The Spares Rack 

557 The final rack in the MINERz/A electronics installation sits across the hall 

558 from the detector on the West side of the enclosure. It contains a number of 

559 spare components: 

560 1. A 3U Linux PC configured to serve as a spare readout node. 

561 2. A 48 port Cisco C2960G switch, designated S-S-MINERVA-2.23 

562 3. A lU Linux PC running logging software for the slow monitoring de- 

563 vices. This machine also serves as the head node for a pool of servers 

564 running a Condor queue for batch processing jobs used to support 

565 monitoring activities in the control room. 

566 4. Spare AC distribution and smoke protection units. 

567 7. 5. Power Infrastructure 

568 7.5.1. DC Power Distribution 

569 The large number of Front End boards and their location atop the de- 

570 tector makes conventional low- voltage power distribution impractical. A dis- 

571 tributed power architecture is employed where 48 VDC power equipment 

572 is located in the electronics racks and intermediate low voltage regulation is 

573 performed on the detector near the FEBs. Finally, low-noise linear regulators 

574 on each FEB produce the final 3.3 V power required. 

575 Each FEB readout chain (up to ten FEBs) is powered by a Front End 

576 Support Board (FESB) which contains an isolated DC-DC converter to step 

577 down 48 VDC to approximately 4 VDC at 20 A. FESBs are passively cooled 

578 and use the FEB/PMT steel mounting bracket as a heat sink. 

579 Three 48 VDC power supplies operate in parallel and support "hot swap" 

580 and dynamic load sharing. Two supplies are required to meet the design 

581 specification of 3000 W. The extra power supply allows for "N+1" redundant 

582 operation, which makes it simple to remove and replace a supply without 

583 interruption. Each power supply is rated for 1900 W and provides an interface 

584 by which the health of the supply may be monitored remotely (through the 

585 Rack Protection System (RPS) interface board and Environmental Monitor). 

586 Through the Fuse Chassis the main 48 VDC 120 A bus is fanned out into 

587 six 20 A connections. Two of these 20 A connections feed a Network Fuse 

588 Panel (NFP), which fans out into twenty 2 A busses going to the FESBs. 

589 Each NFP oflFers independent control and monitoring of the FESBs through 

590 a web interface. 
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591 The actual power supply consumption is 1500 W when the FEBs are 

592 idle. The redundant "N+1" power supply configuration has been robust and 

593 reliable and has operated continuously without problems since commissioning 

594 in 2008. See Fig. [13] for a schematic. 
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Figure 13: Power distribution for MINERi/A. 



595 1.5.2. Rack Protection System and AC Power Distribution 

596 Rack protection is provided by the BiRa Systems RPS unit model 8884^-, 

597 which monitors a smoke detector in each of the three racks on the platform. 

598 If smoke is detected, the RPS unit drops the interlock signal, disabling the 

599 AC distribution chassis. The RPS also generates an emergency power output 

600 (EPO) signal that immediately disconnects the internal batteries on the UPS. 

601 The Environmental Monitor reads various temperature sensors located 

602 around the detector. In addition, this monitor records and logs humidity and 

603 ambient light conditions. It is accessed through a web and FTP interface on 

604 the DAQ network. 

605 8. Interface to the MINERi/A Control Room 

606 8.1. The UmERu A Control Room 

607 The MINERz/A control room is located in Wilson Hall at FNAL, in the 

608 north-west corner room on the 12th floor (WH12). See Fig. [14] for a map 

609 of building locations at Fermilab. MINERz/A services our control room with 
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610 a total of eight PCs, with four in the Control Room and four in the Feyn- 

611 man Computing Center (FCC) at FNAL. The PCs kept in FCC run more 

612 processor-intensive tasks using a private Condor batch queue system—. 




Figure 14: Buildings and sites at Fermilab. 

613 8.2. Controlling the DAQ System 

614 As noted above in section 16. 2[ the DAQ system is controlled via a graphi- 

615 cal thin client delivered with the run control software. The client runs locally 

616 on the operator's terminal. Control of the DAQ is via a custom TCP thin- 

617 client protocol that is transmitted over a Kerberos-enabled SSH tunnel to 

618 the master DAQ node. Inbound connections are filtered both via the firewall 

619 provided by the NuMI router and by an access list of Kerberos principals 

620 allowed to log into the machine. 

621 Commands sent via a successful tunnel to the DAQ manager process 

622 (running on the DAQ master node) are interpreted based on a notion of client 

623 "control" of the DAQ. At most one (though possibly zero) client (s) can be in 

624 control of the DAQ at any given time; only commands that originate from a 

625 node which is currently "in control" are ultimately acted upon. An unlimited 
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626 u number of clients are allowed to connect as observers. Any observer is 

627 allowed to request control at any time; however, if another client is in control 

628 at the time the request is made, the client in control is given a configurable 

629 length of time to veto the request or approve transfer of control. In addition, 

630 the run control package contains a client manager program that can revoke or 

631 assign control to or from any client that is currently connected after proper 

632 credentials are submitted, currently a password. 

633 Once connected and in control, the operator is presented with a graphi- 

634 cal interface containing details about the current data acquisition sequence 

635 and status as well as controls allowing him/her to start, manually advance, 

636 or stop data acquisition. When the DAQ is stopped the operator can also 

637 adjust a handful of simple configuration options pertaining to the data ac- 

638 quisition sequence he/she would like to begin. The bulk of the configuration 

639 operations, however, is accessible only to experts directly on the DAQ head 

640 node so as to avoid accidental misconfiguration. 

641 Once initiated, data acquisition requires no intervention from the oper- 

642 at or. The run control system can even operate unsupervised by any clients 

643 at all (as would happen, for example, in the case where the network connec- 

644 tion to a remote site is interrupted). Under these circumstances warnings 

645 about the system state are queued and delivered in bulk to every client who 

646 reconnects until they are acknowledged. 

647 9. Data Handling Paths 

648 The MINERz/A electronics are capable of monitoring a wide variety of 

649 configuration states: high voltage on the (Cockcroft- Walton) PMTs deployed 

650 on the detector, whether individual boards are dead or live, and the temper- 

651 ature of the FEBs. A separate system monitors the resistive base PMTs used 

652 for the MINERz/A Veto Wall. For cost-saving reasons MINERz/A uses the 

653 same communications chain for electronics configuration and slow controls 

654 as it does for data handling. As a consequence, these monitoring data are 

655 packaged with the physics data directly and exist as part of that stream. 

656 For monitoring purposes, the MINERz/A near-online, or "Nearline," clus- 

657 ter runs an ET client that is notified about the beginning and end of runs. 



^Unlimited at least in principle. Network bandwidth and CPU time saturation, of 
course, will create an operational ceiling, though the actual maximum will depend on the 
environmental conditions. 
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During a run, it reads data from the DAQ head node frame-by-frame. When 
it completes an event (signified by reading the DAQ event header frame) 
it converts the binary data into MINERz/A "RawDigits" (the most basic 
analysis data format) for monitoring. Other tasks running on the Nearline 
cluster decode the RawDigits and process them through the MINERz/A soft- 
ware framework - producing monitoring quantities ranging from high voltage 
status to event displays. These are published to control room PCs for exper- 
imenters to study. 



1) The master node arms the 
detector to receive data. 



2) Signals from the Main 
Injector indicate protons on 
target and we open a gate. 



3) Data are collected at the 
end of each gate by the 
readout nodes. 



Master & Event Builder 
mnvonlinemaster.fnal.gov 



Upstream DAQ 
mnvonlinel .fnal.gov 



Downstream DAQ 
mnvonlinel .fnal.gov 



Control Room 
minerva-om.fnal.gov, etc. 



8) Monitoring data and 
histograms are pulled and made 
available for display. 



6) At end of run, the memory- 
mapped file is converted into a 
flat binary file. 



5) The master node assembles 
the frames and adds an event 
header and writes the data 
into a memory-mapped file. 



4) Each frame of data is 
passed to the master node 
after it is recorded with event 
and type headers attached. 



Near-Online Analysis 
mnvnearlinel .fnal.gov 



7) Data are pulled event by event for 
monitoring. Whole runs are pushed for archival. 



Figure 15: Flow-chart diagram for physics data through the MINERi^A acquisition chain. 
Not shown are the archival data handling beyond the near-online cluster or the metadata 
handling. 



666 After each run a copy of the raw binary data file is pushed to the disk 

667 and tape for permanent storage. See Fig. [15] for a flowchart of the scheme. 

668 A long-term rolling copy of recent raw data is kept on the DAQ system itself 

669 (extending back approximately two months). With the raw data the DAQ 

670 additionally pushes metadata about the run for entry into SAM. Finally, a 

671 "keep-up" processing that uses raw data copied directly from the DAQ (not 

672 from the Nearline system) is also run to produce the RawDigits of record. In 

673 the end, there are three copies of the "raw" data at the end of this process: 

674 a copy of the raw data on the DAQ storage disks, a copy of the raw data in 
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675 permanent storage, and a copy of the RawDigits (raw data reformatted for 

676 MINERz/A software consumption) for general use. 

677 10. Summary and Performance 




Figure 16: Protons on target delivered and recorded for MINERi^A. 

678 MINERz/A began its official physics run on March 23, 2011 with the DAQ 

679 system described above. Since that time, live-time for the whole period was 

680 96.4% (not counting beam down-times). See Fig. [16] for a summary of the 

681 protons on target delivered and recorded for MINERz/A. During normal op- 

682 erations the live-time is over 99% and inefficiencies are almost entirely due 

683 to change-over between runs. Our integrate live-time is most impacted by 

684 electronics failures and simple operator error or inattention. There are ideas 

685 for removing losses due to run change-overs (pre-starting the subsequent run 

686 and putting it into a wait state to speed up the change) and our run control 

687 system is constantly being improved to minimize the opportunity for exper- 
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688 imenters to make mistakes when running the system while still providing 

689 flexible operational control. 
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